What Works Clearinghouse™ 


U.S. DEPARTMENT OF EDUCATION 



WWC Single Study Review 

A review of the design and summary of findings for an individual study 



INSTITUTE of 

EDUCATION SCIENCES 


July 2015 


WWC Review of the Report “Interactive Online Learning on 
Campus: Testing MOOCs and Other Platforms in Hybrid Formats 
in the University System of Maryland ” 12 

The findings from this review do not reflect the full body of research evidence on interactive 

online learning in a hybrid format. 


What is this study about? 

This study measured the impact of using hybrid 
forms of interactive online learning in seven under- 
graduate courses across seven universities in the 
University System of Maryland. Interactive online 
learning can involve video lectures, opportunities 
for discussion and interaction with instructors and 
peers, and online assignments and exams. Hybrid 
forms of such courses combine online learning 
components with traditional face-to-face instruction. 

Of the 1 ,598 students enrolled in large introductory 
biology, statistics, precalculus, computer science, 
and communications courses, 778 were enrolled in 
sections that used the hybrid delivery format, and 
820 were enrolled in the sections that used a tra- 
ditional delivery format. The hybrid delivery format 
entailed one of two types of technology platforms: 
Massive Open Online Courses (MOOCs) supported 
by Coursera or materials from the Open Learning 
Initiative (OLI) developed by Carnegie Mellon Univer- 
sity. Five of the hybrid courses had reduced face- 
to-face class time, and two courses used the online 
materials as supplements. 

The authors measured the impact of the hybrid 
instruction on learning outcomes (course pass rates, 
student grades, and on exam questions that were 
common across the intervention and comparison 
conditions), controlling for SAT scores, student 


GPA, family income, and other student background 
characteristics. In particular, the authors examined 
whether the students participating in the hybrid 
courses performed as well as or better than stu- 
dents in traditional courses. 3 


WWC Rating 


The research described in this 
report does not meet WWC 
group design standards 

Because students were not randomly assigned to the 
intervention and comparison groups, and pretests on 
the outcomes were not available, the study needed 
to demonstrate equivalence on baseline measures of 
socioeconomic status and prior achievement to meet 
WWC group design standards with reservations. 
Baseline information on parents’ income and 
students’ high school GPA was reported in the 
study; however, the authors imputed some missing 
values for analysis. Currently, WWC standards do not 
allow imputed data to be used to establish baseline 
equivalence in quasi-experimental studies. Therefore, 
this study does not meet WWC group design 
standards, and the findings are not presented in this 
WWC report. 
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What did the study find? 

None of the analyses presented in this study meet 
WWC standards, and therefore, the study findings 
are not presented in this WWC report. 


Features of Interactive Online Learning in 
Hybrid Formats 


“Interactive online learning in hybrid formats” means 
incorporating online technology when delivering 
content to students. 

The “hybrid format” designation encompasses any 
combination of face-to-face and online delivery, 
including supplementing face-to-face instruction 
with online material. 

Interactive online learning programs often have 
features that allow teachers to collect and generate 
descriptive data about student learning, in particular 
where students may be struggling and how quickly 
students move through material. This data provides 
feedback for both students and instructors to 
facilitate course improvement. 

The interactive online learning in hybrid formats 
examined in this study included math, statistics, 
biology, computer science, and communications. 

Some of examples of the hybrid delivery used in this 
study included: 

• Using Massively Open Online Courses (MOOCs) 
or materials from the Open Learning Initiative (OLI) 
to replace textbooks with an online version, 

• Developing additional material in a MOOC to 
supplement in-class instruction (i.e., using a 
MOOC on critical reasoning to teach students how 
to approach a concept and evaluate information 
resources more analytically), and 

• Enhancing an online course with video lectures or 
a MOOC to convey information in a particular area 
not in an instructor’s area of expertise. 
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Endnotes 

1 Griffiths, R., Chingos, M., Mulhern, C., & Spies, R. (2014). Interactive online learning on campus: Testing MOOCs and other platforms 
in hybrid formats in the University System of Maryland. New York, NY: Ithaka S+R. 

2 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors) to assess whether the study design meets WWC design standards. The review reports the WWC’s assessment of whether 
the study meets WWC design standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Individual Studies in the Postsecondary Education topic area review protocol (ver- 
sion 3.0). A quick review of this study was released on August 1 1 , 2014, and this report is the follow-up review that replaces that initial 
assessment. 

3 There were two outcomes included in the study that are not described in this WWC report. The study reported qualitative findings 
about implementation and faculty satisfaction; however, these data were not included in the review per guidelines set forth in the 
WWC Procedures and Standards Handbook (version 3.0). 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2015, July). 

WWC review of the report: Interactive online learning on campus: Testing MOOCs and other platforms in 
hybrid formats in the University System of Maryland. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 


Practice 

Guide 


Quick Single Study 
Review Review 



A single study review of an individual study includes the WWC’s assessment of the quality of the research design 
and technical details about the study’s design and findings. 

This single study review was prepared for the WWC by Development Services Group, Inc. under contract ED-IES-12-C-0084. 
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